Modular on-line function approximation for scaling up reinforcement learning

Reinforcement l e a r n i n g i s a p o werful learning paradigm for autonomous agents which i n teract with unknown environments with the objective of maximizing cumulative p a yoo. Recent research has addressed issues concerning the scaling up of reinforcement learning methods in order to solve problems with large state spaces, composite tasks and tasks involving non-Markovian situations. In this dissertation, I extend existing ways of scaling up reinforcement learning methods and propose several new approaches. An array of Cerebellar Model Articulation Controller (CMAC) networks is used as fast function approximators so that the evaluation function and policy can be learnt on-line as the agent i n teracts with the environment. Learning systems which combine reinforcement learning techniques with CMAC networks are developed to solve problems with large state and action spaces. Actions can be either discrete or real-valued. The problem of generating a sequence of torque or position change commands in order to drive a simulated multi-linked manipulator towards desired arm conngurations is examined. A hierarchical and modular function approximation architecture using CMAC n e t works is then developed, following the Hierarchical Mixtures of Experts framework. The non-linear function approximation ability o f C M A C n e t works enables non-linear functions to be modelled in expert and gating networks, while permitting fast linear learning rules to be used. An on-line gradient ascent learning procedure derived from the Expectation Maximization algorithm is proposed, enabling faster learning to be achieved. The new architecture can be used to enable reinforcement learning agents to acquire context-dependent evaluation functions and policies. This is demonstrated in an implementation of the Compositional Q-Learning framework in which composite tasks consisting of several elemental tasks are decomposed using reinforcement learning. The framework is extended to the case where rewards can be received in non-terminal states of elemental tasks, and tòvector of actions' situations where the agent produces several coordinated actions in order to achieve a goal. The resulting system is employed to enable the simulated multi-linked manipulator to position its end-eeector at several positions in the workspace sequentially. Finally, the beneets of using prior knowledge in order to extend the capabilities of reinforcement learning agents are examined. A classiier system-based Q-learning scheme is developed to enable agents to reason using condition-action rules. The utility o f t h i s s c heme is illustrated in a …